WEBVTT Kind: captions Language: en 00:00:00.970 --> 00:00:03.640 Hello and welcome back to another exciting episode 00:00:03.640 --> 00:00:06.600 of Game On!, our show for game developers. 00:00:06.600 --> 00:00:08.440 I'm your host Todd Kerpelman. 00:00:08.440 --> 00:00:10.460 I know it's been awhile since our last episode 00:00:10.460 --> 00:00:11.394 and I've missed you. 00:00:11.394 --> 00:00:13.310 Did you do something different with your hair? 00:00:13.310 --> 00:00:15.090 It looks nice. 00:00:15.090 --> 00:00:16.950 Anyway, this episode is all about not 00:00:16.950 --> 00:00:18.390 shipping broken games. 00:00:18.390 --> 00:00:21.280 Or I guess to put a more positive spin on it, 00:00:21.280 --> 00:00:23.530 testing your game for fun and profit. 00:00:23.530 --> 00:00:24.784 Mostly profit. 00:00:24.784 --> 00:00:27.200 And it's all done through the magic of using Google Play's 00:00:27.200 --> 00:00:29.590 alpha and beta services to get your game out 00:00:29.590 --> 00:00:32.372 in front of a large audience before you publish. 00:00:32.372 --> 00:00:34.080 So I'd like to start today by telling you 00:00:34.080 --> 00:00:35.760 a little story about Don. 00:00:35.760 --> 00:00:37.520 See Don is a game developer and he just 00:00:37.520 --> 00:00:40.450 released the very first version of his hotly anticipated game 00:00:40.450 --> 00:00:41.200 to the world. 00:00:41.200 --> 00:00:43.766 He was so excited, he could barely sleep that night. 00:00:43.766 --> 00:00:46.390 And so when he woke up the next morning, the first thing he did 00:00:46.390 --> 00:00:48.540 was go out and check his apps review page, which 00:00:48.540 --> 00:00:51.110 looked a little something like this. 00:00:51.110 --> 00:00:52.250 Huh? 00:00:52.250 --> 00:00:53.630 What happened? 00:00:53.630 --> 00:00:55.920 Well, Don did some digging around 00:00:55.920 --> 00:00:58.810 and he found out that due to a font rendering error, 00:00:58.810 --> 00:01:02.430 the game crashed for any player who had an E in their name. 00:01:02.430 --> 00:01:04.000 Well Don immediately got together 00:01:04.000 --> 00:01:08.000 with his team members, Alicia Winkowski, Irwin Lin, and Chris 00:01:08.000 --> 00:01:09.772 Fulbright and said, how did we miss this? 00:01:09.772 --> 00:01:11.730 How could we have possibly overlooked something 00:01:11.730 --> 00:01:14.195 as completely obvious as-- OK, wait I think I see it. 00:01:14.195 --> 00:01:15.570 None of us have an E in our name. 00:01:15.570 --> 00:01:17.310 That's kind of a problem. 00:01:17.310 --> 00:01:19.010 That's when he thought, huh, maybe we 00:01:19.010 --> 00:01:21.218 should have tested this with a bigger group of people 00:01:21.218 --> 00:01:23.100 before we launched this thing. 00:01:23.100 --> 00:01:25.792 But of course, that sets up its own share of problems, right? 00:01:25.792 --> 00:01:27.500 How are you going to be sharing your game 00:01:27.500 --> 00:01:29.416 with several dozen friends and family members? 00:01:29.416 --> 00:01:31.544 Are you going to be mailing APKs around? 00:01:31.544 --> 00:01:33.960 How are you going to make sure that only these people have 00:01:33.960 --> 00:01:34.460 access? 00:01:34.460 --> 00:01:36.251 How are you going to make sure that they're 00:01:36.251 --> 00:01:37.510 looking at the latest version? 00:01:37.510 --> 00:01:40.480 Do you have to mail them and bug them each time you update 00:01:40.480 --> 00:01:43.787 and hope that they then download and resideload your APK? 00:01:43.787 --> 00:01:45.370 Come to think of it, how are you going 00:01:45.370 --> 00:01:46.890 to teach your nontechnical users how 00:01:46.890 --> 00:01:48.910 to sideload an APK in the first place? 00:01:48.910 --> 00:01:50.524 It can be a bit of a mess. 00:01:50.524 --> 00:01:53.190 And that brings us to one of the best features of the Play Store 00:01:53.190 --> 00:01:55.125 publishing process that some of you 00:01:55.125 --> 00:01:57.700 are overlooking, which frankly is a real shame because it 00:01:57.700 --> 00:02:00.030 was built for just this situation. 00:02:00.030 --> 00:02:03.250 And that is our Alpha Beta Staged Rollout process. 00:02:03.250 --> 00:02:04.890 Also sometimes known as Alpha Beta 00:02:04.890 --> 00:02:08.350 Canary because that makes for a better looking acronym. 00:02:08.350 --> 00:02:11.260 This lets you easily test out new apps and app updates 00:02:11.260 --> 00:02:13.230 with large groups of people before you 00:02:13.230 --> 00:02:14.910 publish to the world. 00:02:14.910 --> 00:02:17.230 So let's see what this process looks like. 00:02:17.230 --> 00:02:18.730 Now in the past you're probably used 00:02:18.730 --> 00:02:20.682 to just uploading your APK into production, 00:02:20.682 --> 00:02:22.390 where it gets published to the Play Store 00:02:22.390 --> 00:02:23.640 and is available to the world. 00:02:23.640 --> 00:02:25.180 And hurray, there's pixie dust. 00:02:25.180 --> 00:02:28.040 Or I suppose, if things go badly, uh, oh my god. 00:02:28.040 --> 00:02:30.380 Everything's on fire, run. 00:02:30.380 --> 00:02:31.230 One of the two. 00:02:31.230 --> 00:02:32.140 Not sure. 00:02:32.140 --> 00:02:33.600 But you don't have to do that. 00:02:33.600 --> 00:02:36.360 So hang on their buddy, stay. 00:02:36.360 --> 00:02:38.080 You see, the Developer Console also 00:02:38.080 --> 00:02:39.990 lets you specify a certain group of people 00:02:39.990 --> 00:02:41.760 to be your alpha testers. 00:02:41.760 --> 00:02:45.160 This could be a Google group or a Google+ community. 00:02:45.160 --> 00:02:47.300 For example, for my first group of alpha testers, 00:02:47.300 --> 00:02:51.210 I might want this just to be the employees in my organization. 00:02:51.210 --> 00:02:53.900 So I could publish-- so I could define my alpha testers 00:02:53.900 --> 00:02:57.864 to be this group and then publish my APK just to them. 00:02:57.864 --> 00:02:59.780 Now assuming that all goes well and we haven't 00:02:59.780 --> 00:03:01.390 found any major bugs, I could then 00:03:01.390 --> 00:03:03.850 create a second group of beta testers, maybe people 00:03:03.850 --> 00:03:04.990 outside my company. 00:03:04.990 --> 00:03:08.030 And again, publish my APK just to them next. 00:03:08.030 --> 00:03:10.270 Kind of similar process as before. 00:03:10.270 --> 00:03:12.074 Now, as for how this looks to your testers, 00:03:12.074 --> 00:03:14.490 you're just going to send them a link to a page where they 00:03:14.490 --> 00:03:17.229 can click on a button to become a tester. 00:03:17.229 --> 00:03:19.270 And once they do that, they can download the game 00:03:19.270 --> 00:03:20.950 from the Play Store just as if it 00:03:20.950 --> 00:03:22.880 were a completely normal app. 00:03:22.880 --> 00:03:24.540 No weird sideloading involved. 00:03:24.540 --> 00:03:27.160 Access is completely handled by your group settings. 00:03:27.160 --> 00:03:29.890 You can leave it open or invite only, however you want. 00:03:29.890 --> 00:03:33.630 And all those concerns Don had, they're basically solved. 00:03:33.630 --> 00:03:35.870 So where do you get these testers? 00:03:35.870 --> 00:03:37.200 Well, here's one fun example. 00:03:37.200 --> 00:03:40.340 The developers of CastleStorm promoted their Android program 00:03:40.340 --> 00:03:41.250 right on their site. 00:03:41.250 --> 00:03:43.090 So when you went and visited their website, 00:03:43.090 --> 00:03:45.600 you would see this page, click on this little green Play 00:03:45.600 --> 00:03:47.490 Now button, took you to their Google group, 00:03:47.490 --> 00:03:49.370 which was open for anybody to join. 00:03:49.370 --> 00:03:52.370 And they got a ton of great beta testers this way. 00:03:52.370 --> 00:03:54.970 But don't forget about your Google+ communities. 00:03:54.970 --> 00:03:57.600 These are often highly engaged fans of your game. 00:03:57.600 --> 00:04:00.160 And they'd be really good candidates for beta testers. 00:04:00.160 --> 00:04:02.600 Not only for any new updates to existing games, 00:04:02.600 --> 00:04:04.760 but also for any new product that you release. 00:04:04.760 --> 00:04:06.620 And by the way, this is a fantastic way 00:04:06.620 --> 00:04:08.260 to connect with your community. 00:04:08.260 --> 00:04:10.700 Getting early feedback on new games or new updates 00:04:10.700 --> 00:04:13.150 from your most dedicated fans will provide you 00:04:13.150 --> 00:04:16.430 with some fantastic insight from people who quite frankly know 00:04:16.430 --> 00:04:18.129 your game probably better than you. 00:04:18.129 --> 00:04:20.670 And it also makes them part of the process and a collaborator 00:04:20.670 --> 00:04:21.971 in your game. 00:04:21.971 --> 00:04:23.970 And then they can become your biggest supporters 00:04:23.970 --> 00:04:26.570 and your biggest advocates in the outside world. 00:04:26.570 --> 00:04:28.690 Your players, they want your games to be awesome. 00:04:28.690 --> 00:04:30.230 They're on your side. 00:04:30.230 --> 00:04:32.450 Don't be afraid to engage with them. 00:04:32.450 --> 00:04:34.516 However you do it, assuming everything goes well 00:04:34.516 --> 00:04:35.890 with your beta test group, you're 00:04:35.890 --> 00:04:37.560 going to push wrap up to production. 00:04:37.560 --> 00:04:40.025 And then the world can finally play your game. 00:04:40.025 --> 00:04:41.777 Now this is all great, but what happens 00:04:41.777 --> 00:04:43.110 if you want to update your game? 00:04:43.110 --> 00:04:44.932 Maybe you've got a round of bug fixes, 00:04:44.932 --> 00:04:47.390 but you want to make sure this bug fix doesn't accidentally 00:04:47.390 --> 00:04:49.930 create a whole bunch of bigger bugs in the process. 00:04:49.930 --> 00:04:52.210 It's been known to happen to some game developers. 00:04:52.210 --> 00:04:56.450 Quite possibly ones in this room giving presentations right now. 00:04:56.450 --> 00:04:58.670 So as you might expect, alpha and beta 00:04:58.670 --> 00:05:01.130 works similar for game updates as well. 00:05:01.130 --> 00:05:03.240 I can upload a new version of my game 00:05:03.240 --> 00:05:05.411 to the alpha group and all my alpha testers, 00:05:05.411 --> 00:05:07.660 they'll be able to grab the latest version I have just 00:05:07.660 --> 00:05:09.800 as if the app have been updated normally for them in the Play 00:05:09.800 --> 00:05:10.300 Store. 00:05:10.300 --> 00:05:12.470 In fact, in many situations, your alpha testers 00:05:12.470 --> 00:05:14.240 will just get the update automatically. 00:05:14.240 --> 00:05:16.531 And then from there, I promote my new version of my app 00:05:16.531 --> 00:05:19.180 to beta, have my beta testers try it out and then push it 00:05:19.180 --> 00:05:20.180 on to production, right? 00:05:20.180 --> 00:05:22.670 Well, wait, stop, stay there one second. 00:05:22.670 --> 00:05:25.370 See this time around, for updates to an existing app, 00:05:25.370 --> 00:05:27.010 there's a shocking twist. 00:05:27.010 --> 00:05:27.760 Yes, that's right. 00:05:27.760 --> 00:05:30.860 Even our stock image photography model is surprised. 00:05:30.860 --> 00:05:33.210 You see, instead of just updating my app fully 00:05:33.210 --> 00:05:34.690 on production, I can do something 00:05:34.690 --> 00:05:36.392 that's called a Staged Rollout. 00:05:36.392 --> 00:05:38.850 What this lets me do is push out the new version of my game 00:05:38.850 --> 00:05:41.480 to just a small percentage of the market. 00:05:41.480 --> 00:05:43.160 This is a great way to make extra sure 00:05:43.160 --> 00:05:45.540 your game is going to work well with a larger audience. 00:05:45.540 --> 00:05:47.310 It also turns out is a nice way to see 00:05:47.310 --> 00:05:49.280 what people who aren't your friends and family 00:05:49.280 --> 00:05:50.579 might think of your game. 00:05:50.579 --> 00:05:52.120 Turns out your mom might not actually 00:05:52.120 --> 00:05:53.775 be your most objective customer. 00:05:53.775 --> 00:05:54.920 Sorry, Mom. 00:05:54.920 --> 00:05:57.200 So what this means is I can have Version 2 of my game 00:05:57.200 --> 00:05:59.360 out to a small fraction of the world while having 00:05:59.360 --> 00:06:01.934 Version 1 of my game available to everybody else. 00:06:01.934 --> 00:06:03.600 And then as time goes on and I'm feeling 00:06:03.600 --> 00:06:05.300 fairly confident about this update, 00:06:05.300 --> 00:06:10.070 I can expand that rollout more and more and more 00:06:10.070 --> 00:06:12.090 until the whole world is finally ready to see 00:06:12.090 --> 00:06:14.154 the update to my game. 00:06:14.154 --> 00:06:15.570 And the nice thing is that you can 00:06:15.570 --> 00:06:19.000 see your reviews, your crashes, and your other stats broken out 00:06:19.000 --> 00:06:22.410 by application version right there in the Developer Console. 00:06:22.410 --> 00:06:24.760 So if your players hate your update, 00:06:24.760 --> 00:06:27.287 you can see that right away in the reviews. 00:06:27.287 --> 00:06:29.620 More importantly, you can also set up your own analytics 00:06:29.620 --> 00:06:31.300 to be broken out by version number. 00:06:31.300 --> 00:06:34.050 So you can see if there are any major problems with retention 00:06:34.050 --> 00:06:36.690 or ARPDAU or ARPU or any other custom 00:06:36.690 --> 00:06:38.470 metrics you happen to be tracking. 00:06:38.470 --> 00:06:40.940 For instance, if we find out that nobody is completing 00:06:40.940 --> 00:06:43.577 the tutorial in our v2 release, well that's 00:06:43.577 --> 00:06:45.410 a sign that something is really wrong there. 00:06:45.410 --> 00:06:48.030 And we can fix that before we push this update out 00:06:48.030 --> 00:06:50.030 to the entire population. 00:06:50.030 --> 00:06:51.614 So this all sounds great right? 00:06:51.614 --> 00:06:53.780 Well, with that said, let me address three responses 00:06:53.780 --> 00:06:56.630 I've heard from developers around Alpha Beta testing. 00:06:56.630 --> 00:06:58.450 The first is, well I'm not sure I 00:06:58.450 --> 00:07:00.783 want to use this because I don't want to miss the chance 00:07:00.783 --> 00:07:02.000 to be a "Top Recent" game. 00:07:02.000 --> 00:07:04.590 And let me explain to you what the concern is here. 00:07:04.590 --> 00:07:08.080 So imagine that you spent a lot of time with your game in Alpha 00:07:08.080 --> 00:07:10.660 and then Beta test stages. 00:07:10.660 --> 00:07:12.790 Then you're game hits production here 00:07:12.790 --> 00:07:14.430 and that's when it takes off. 00:07:14.430 --> 00:07:17.480 Well, do we qualify as the "Top Recent" game? 00:07:17.480 --> 00:07:20.090 We have spent a couple of months in Alpha and Beta. 00:07:20.090 --> 00:07:21.190 And it's actually true. 00:07:21.190 --> 00:07:24.390 We used to start the "Top Recent" time here, 00:07:24.390 --> 00:07:27.200 when your game first pushed to Alpha, which I think we all 00:07:27.200 --> 00:07:28.750 agree makes no sense. 00:07:28.750 --> 00:07:30.220 So you'll be happy to know we now 00:07:30.220 --> 00:07:32.470 start the "Top Recent" timer here, 00:07:32.470 --> 00:07:34.920 once your game has been pushed to production. 00:07:34.920 --> 00:07:37.820 So if you hear of anybody who's still afraid to Alpha test 00:07:37.820 --> 00:07:40.290 because they're worried they're going to miss their chance 00:07:40.290 --> 00:07:41.990 to be on the "Top Recent" list, let 00:07:41.990 --> 00:07:43.570 them know we've addressed this issue. 00:07:43.570 --> 00:07:45.210 They're working on old info and they 00:07:45.210 --> 00:07:47.460 don't need to be concerned about this anymore. 00:07:47.460 --> 00:07:50.320 And in fact, what I'm also excited to tell you-- and go 00:07:50.320 --> 00:07:51.780 grab your business analyst person 00:07:51.780 --> 00:07:53.470 and tell them to watch this part-- 00:07:53.470 --> 00:07:55.840 is that we are now calculating the "Top Recent" list 00:07:55.840 --> 00:07:57.020 per country. 00:07:57.020 --> 00:07:59.320 So if you want to launch your product in say, Canada 00:07:59.320 --> 00:08:02.300 first and then launch two weeks later in the US, 00:08:02.300 --> 00:08:04.000 your "Top Recent" window in the US 00:08:04.000 --> 00:08:06.100 will be determined by the time you first 00:08:06.100 --> 00:08:08.340 launch there and not in Canada. 00:08:08.340 --> 00:08:10.200 And so you can launch in different countries 00:08:10.200 --> 00:08:12.240 at different times and not worry about missing 00:08:12.240 --> 00:08:13.657 the "Top Recent" window elsewhere. 00:08:13.657 --> 00:08:16.031 See, your business analyst person, they're really excited 00:08:16.031 --> 00:08:16.755 now aren't they? 00:08:16.755 --> 00:08:18.910 They're glad you grabbed them. 00:08:18.910 --> 00:08:20.410 So a second question I sometimes get 00:08:20.410 --> 00:08:23.344 is how do you add achievements or leaderboards or game 00:08:23.344 --> 00:08:26.010 services-- other game services-- to a version of the game that's 00:08:26.010 --> 00:08:27.080 in beta? 00:08:27.080 --> 00:08:29.300 And it turns out these days it's pretty easy. 00:08:29.300 --> 00:08:31.300 When you create a new leaderboard or achievement 00:08:31.300 --> 00:08:34.190 or quest or event or what have you in Play game services, 00:08:34.190 --> 00:08:36.980 it remains in an unpublished state on our servers 00:08:36.980 --> 00:08:38.900 that's only visible to your testers. 00:08:38.900 --> 00:08:41.830 But if you go into the testers portion of the Play Services 00:08:41.830 --> 00:08:43.940 Developer Console, right here, you 00:08:43.940 --> 00:08:47.200 can see that you can easily add alpha or beta testers as part 00:08:47.200 --> 00:08:49.870 of the group that has access to these unpublished features 00:08:49.870 --> 00:08:51.110 on our servers. 00:08:51.110 --> 00:08:53.950 So if you're new APK has say a new leaderboard and five 00:08:53.950 --> 00:08:57.050 new achievements, you can simply publish that updated client 00:08:57.050 --> 00:08:58.230 to your Alpha group. 00:08:58.230 --> 00:09:01.282 And then, give those same alpha testers access 00:09:01.282 --> 00:09:03.240 to this new achievement and the new leaderboard 00:09:03.240 --> 00:09:05.370 and achievements on the server. 00:09:05.370 --> 00:09:07.329 Your testers get to test these new services out 00:09:07.329 --> 00:09:09.869 and then the general public will see this mystery leaderboard 00:09:09.869 --> 00:09:11.550 or these mystery achievements that they 00:09:11.550 --> 00:09:14.140 can't seem to access or unlock. 00:09:14.140 --> 00:09:15.980 A third question I sometimes get is, OK, 00:09:15.980 --> 00:09:18.320 so is this what I should be using for A/B testing? 00:09:18.320 --> 00:09:22.400 And my answer is kind of, sort of, but not really, it depends. 00:09:22.400 --> 00:09:25.750 Don't be confused by these A/B letters. 00:09:25.750 --> 00:09:28.570 Alpha Beta testing is different than the split testing people 00:09:28.570 --> 00:09:31.150 sometimes refer to as A/B testing. 00:09:31.150 --> 00:09:32.920 The point of Alpha Beta Canary testing 00:09:32.920 --> 00:09:34.336 is really to find out if there are 00:09:34.336 --> 00:09:36.337 major problems with your game before you launch. 00:09:36.337 --> 00:09:38.086 Now you could certainly use it to find out 00:09:38.086 --> 00:09:40.190 if you've accidentally broken your in-game economy 00:09:40.190 --> 00:09:42.400 or if some new feature has improved retention. 00:09:42.400 --> 00:09:45.230 But if you're looking to A/B test small interface changes 00:09:45.230 --> 00:09:47.600 or gameplay tweaks or the wording 00:09:47.600 --> 00:09:49.680 that you've put in your purchase dialogue, 00:09:49.680 --> 00:09:50.930 it's not really best for that. 00:09:50.930 --> 00:09:53.770 You don't want your users constantly downloading new APKs 00:09:53.770 --> 00:09:55.800 and using up their data plan every time 00:09:55.800 --> 00:09:58.127 you want to test some minor gameplay tweak. 00:09:58.127 --> 00:09:59.710 So instead, I highly recommend looking 00:09:59.710 --> 00:10:02.350 at tools like Google Tag Manager, which basically allow 00:10:02.350 --> 00:10:05.040 you to configure your game and run all the A/B testing you 00:10:05.040 --> 00:10:06.260 want in the cloud. 00:10:06.260 --> 00:10:08.160 Google Tag Manager is a really great tool. 00:10:08.160 --> 00:10:09.940 I am a fan, but I probably don't have time 00:10:09.940 --> 00:10:10.856 to talk about it here. 00:10:10.856 --> 00:10:13.640 I'll have to cover that in a future episode. 00:10:13.640 --> 00:10:16.320 So with all of that in mind, Don, our intrepid game 00:10:16.320 --> 00:10:19.260 developer, got into his handy dandy time machine 00:10:19.260 --> 00:10:22.240 and went back in time to the week before his game launched. 00:10:22.240 --> 00:10:24.651 This time, instead of just testing his game in house, 00:10:24.651 --> 00:10:27.150 he decided to launch it on the beta channel to several dozen 00:10:27.150 --> 00:10:28.690 of his closest friends. 00:10:28.690 --> 00:10:31.500 And that's where he found and fixed several other bugs 00:10:31.500 --> 00:10:34.179 that got missed by his three person team. 00:10:34.179 --> 00:10:35.970 With those bugs fixed, he launched his game 00:10:35.970 --> 00:10:37.140 to great fanfare. 00:10:37.140 --> 00:10:38.260 The reviews were way up. 00:10:38.260 --> 00:10:39.926 And just a few months later, his company 00:10:39.926 --> 00:10:42.499 was acquired for $12 billion. 00:10:42.499 --> 00:10:44.790 Not because of the game though, but because of the time 00:10:44.790 --> 00:10:45.320 machine. 00:10:45.320 --> 00:10:47.850 Turns out those things, they're really valuable. 00:10:47.850 --> 00:10:50.100 Anyway, if you are not in the time machine business, 00:10:50.100 --> 00:10:51.770 but you are in the game business, 00:10:51.770 --> 00:10:53.561 I'm hoping I've convinced you at this point 00:10:53.561 --> 00:10:56.410 that Alpha Beta Canary testing is 00:10:56.410 --> 00:10:57.660 something you should be doing. 00:10:57.660 --> 00:10:59.618 It's generally pretty easy to get started with. 00:10:59.618 --> 00:11:01.540 It's all part of the Play Developer Console. 00:11:01.540 --> 00:11:03.190 But if you want to see a walkthrough 00:11:03.190 --> 00:11:05.220 or see me record a screen capture or something, 00:11:05.220 --> 00:11:06.386 let me know in the comments. 00:11:06.386 --> 00:11:07.150 I'll do that. 00:11:07.150 --> 00:11:10.180 So that's it for this Game On! episode on testing your app. 00:11:10.180 --> 00:11:11.200 Thanks for tuning in. 00:11:11.200 --> 00:11:14.230 Test your games with your family and friends and fans 00:11:14.230 --> 00:11:16.790 and let's start seeing some great games out there. 00:11:16.790 --> 00:11:18.334 Bye.